Fibre Channel is the dominant protocol today for connecting Storage Area Networks (SAN). There are other protocols such as FICON that have the same physical layer interface as Fibre Channel and can be transported using the methods described here. The Fibre Channel protocol enables transmission of high-speed signals across geographically dispersed computers and storage systems.
Traditionally, file servers with large external disks or disk farms using the SCSI standard have been used to support applications requiring large amounts of data storage. As applications increased, the storage system capacities and bandwidth (data transfer speed) requirements increased. The SCSI standard limitations made scaling difficult. The servers could only access data on devices directly attached to them. Failure of the server or SCSI hardware could cause an access failure. Also, SCSI supports only a finite number of devices and is therefore not scalable. The parallel structure of SCSI results in distance limitations that require equipment to be co-located.
Storage Area Networks (SAN) were implemented to overcome the limitations of the SCSI architecture. The SAN is a network between the servers and the storage devices. A SAN allows multiple servers to access any storage device. This increases fault tolerance and overcomes the distance limitation since the server and storage do not need to be co-located. The dominant networking technology for implementing SAN is Fibre Channel.
Fibre Channel technology [ANSI X3T11] was designed to enable high-speed data transfer between computer systems and storage devices. It supports common transport protocols including Internet Protocol and SCSI. It supports high-speed data transfer at standard rates of 1 Gbps, 2 Gbps, 4 Gbps, 8 Gbps, and 10 Gbps. It also supports communications across extended distances enabling corporations to have off-site storage thus enabling applications like disaster recovery and business continuity.
Fibre Channel data is comprised of frames that are concatenated into sequences to enable block transfers. The frame size may vary from 36 B to 2 KB. An exchange can consist of multiple sequences permitting large data transfers. As much as a 128 MB can be transported with a single command. The maximum amount of data that can be in flight across the link is dependent on the buffer credits. The buffer credits define the number of frames that are available to store the data in the event of any blockage along the data link. If the receiving equipment throttles data flow, the buffer must store the data that is in flight while the flow control mechanism provides backpressure to stop the traffic. Therefore, the amount of buffering needed is in excess of the round trip time of the communication system.
The amount of distance extension possible is a function of the amount of buffer memory that can be accommodated within the optical transport system. The buffer memory amount accommodated is a function of the size, power consumption, and density. High-speed memory is required so that it can operate at the bandwidth of the system. It must occupy small physical space so that it can be embedded in the transport equipment. The power consumption must also be within the constraints of the system. Thus, the amount of memory that can be accommodated within the physical constraints of the system defines the geographical distance extension possible.
Usually, SAN are connected to each other via fibre channel switches that have limited distance capability. Most fibre channel switches have limited buffer credits and limit SAN distance to 100 km. The Fibre Channel standard itself has a limitation of 250 km. Most equipment falls within the 250 km and is nominally capable of 100 km distances. Furthermore, the switches are optimized to communicate with themselves often running proprietary traffic. Therefore, a data transparent SAN extension method is often desired for interoperability with other equipment.
Embedding the distance extension within the optical transport system reduces overall cost, increases security, improves reliability, and results in increased throughput. Traditionally, the distance extension has been over a public network over Ethernet or SONET. This method results in decreased performance, reduced security, and increased costs. Accumulation of bit errors can degrade the throughput of the network. Connection through public networks increases vulnerability to attacks or increases costs since encryption devices are needed to provide secure communications.
Nonzero packet loss rates can also severely impact the throughput of FC/FICON transport. Public IP-based networks and even SONET private lines can introduce an error rate that forces significant end-to-end retransmissions for lost data. As distance between data centers increases, overall throughput and synchronization decreases from the effect of retransmissions at the FC/FICON upper-layer protocols (ULPs).
Fibre Channel-over-IP (FCIP) solutions that use shared switched or routed networks also suffer from increased latency from intermediate routing and switching sites. The forced TCP layer retransmissions due to packet loss also require significant bandwidth over allocation due to the drastic reduction in effective throughput. The security issues in public IP-based networks also require additional security measures that are not required for dedicated private networks. Enterprises thus resort to expenditures for additional encryption gear for their FCIP traffic. The use of many disparate boxes to provide the storage extension function and security result in an overall increase in cost, physical space required, and power consumption.
Traditional FC-over-SONET extension utilizes channel extenders or SAN extension gateways that access a traditional carrier network through a SONET access link. The end-to-end SONET connection traverses multiple SONET links across metro carrier and inter-exchange carrier (IXC) networks. Each SONET link is prone to a certain bit error rate (BER), without any error correction scheme such as Forward Error Correction (FEC) employed. Furthermore, the BER of the end-to-end SONET connection accumulates the bit errors across multiple links.
All of these considerations indicate a need for the fiber channel distance extension solution to be a part of the optical transport system. The solution described provides the security of a private network for institutions that transport financial and other critical data. Bypassing traditional public networks also improves communications reliability and results in increased throughput. This architecture also eliminates the need for additional equipment for SONET/Ethernet conversion and data encryption/decryption. This reduces the overall cost for applications such as disaster recovery and business continuity.
The equipment described incorporates the fibre channel extension function within the optical transport system using high-speed memory and proprietary flow control. The QDR SRAM memory used in this application provides the high density, low power consumption, and speed required to buffer the data and provide the distance extension. The flow control method improves upon fibre channel flow control and enables efficient use of memory for all packet sizes without loss of throughput. The concept of embedding the Fibre Channel data within the transport equipment provides the security of a private network. Use of forward error correction (FEC) to connect SAN improves the overall reliability of the link and results in increased throughput.
Due to the large sizes of modern databases, it is also desirable to have a plurality of high-speed data channels on the same fiber to provide scalable bandwidth. The capacity of modern storage systems is increasing beyond 500 Terabytes. The databases may carry financial and other data. More than 10 percent of the data can change over the course of a business day. If 50 Terabytes (400 Terabits) of data changes over 8 hours, 14 Gbps of bandwidth is required. This assumes that the data is changing at a constant rate. However, during the course of a business day, the data changes may occur in bursts so the peak data rate is much higher. Therefore, the network must be designed for the peak data rate to accommodate bursts of traffic. Criteria for peak to average vary depending on the traffic type. The average to peak ration may vary from 12.5% average to peak to 50%. In this example, 1.4 wavelengths required at the constant rate may increase to 2.8 (50%) or 9.2 (12.5%). As the storage capacity increases due to new higher capacity systems entering the market place or addition of parallel disk systems, the bandwidth requirement will increase accordingly. As higher data rate fiber channel interfaces are developed, the transport system must also support these new standards. Thus, a fiber channel distance extension solution with flexible interfaces and scalable capacity is required.
The apparatus and method shown enables fibre channel distance extension beyond 6000 km. High-density, low power consumption memory technology is embedded in each channel of the optical transport system to provide a large buffer for fibre channel data and thus extend the distance of SAN. A non-intrusive fibre channel extension method is used that provides data transparency without terminating the fibre channel traffic. Furthermore, the transport equipment is transparent to the terminating SAN equipment in that no Fibre Channel LOGIN is required into the transport equipment. In essence, the end devices are talking to each other. The distance extension is accomplished with fibre channel in its native mode without conversion to SONET or IP. Flexible fibre channel aggregation and extension architecture is used to allow flexible interfaces and scalable bandwidth. This enables aggregation of multiple low speed fiber channel interfaces to a single high-speed data channel.
Methods for sending data signals are taught in U.S. Pat. No. 6,151,334 to Kim, et al, United States Patent Publication No. 2002/0080809 to Nicholson, et al, United States Patent Publication No. 2002/0075903 to Hind and U.S. Pat. No. 6,396,853 to Humphrey et al.